{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2c002fc4-85b9-4c1a-b4b5-0709baea1f01",
   "metadata": {},
   "source": [
    "# FastText算法复现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82e7d60e-7358-4962-9e38-97b819eefa52",
   "metadata": {},
   "source": [
    "FastText 是由 Facebook 提出的词向量模型，它在 Word2Vec 的基础上进行了改进。主要的改进点在于：FastText 不仅考虑整个单词，还考虑单词内部的字符 n-gram 信息。这种方法使得 FastText 在处理形态丰富的语言（如法语、德语等）时表现更加出色，并且能够生成更细粒度的词向量。\n",
    "\n",
    "## 一、概述\n",
    "尽管深度学习神经网络在自然语言处理方面表现出色，但它们通常需要数十层甚至上亿个参数，因此速度较慢且需要大量计算资源的支持，这限制了它们的应用场景。相比之下，fastText是一种由Facebook开源的简单而高效的文本分类器，采用浅层神经网络实现了word2vec和文本分类功能。fastText与深度网络在效果上几乎相当，但更加节省资源，并且速度提高了百倍，因此可以被视为高效的工业级解决方案。\n",
    "\n",
    "fastText是在[负采样的Skip-Gram模型](https://aistudio.baidu.com/aistudio/projectdetail/5406915?contributionType=1&sUid=2554132&shared=1&ts=1678689283700)基础上，进一步的将每个中心词视作子词的集合，并学习该子词的词向量。\n",
    "\n",
    "\n",
    "\n",
    "简单理解该方案：\n",
    "\n",
    "\n",
    "\n",
    "以like这个词为例，设子词为2个字符，则子词包括\"<l\" , \"li\",\"ik\",\"ke\", \"e>\" 和特殊子词\"\\<like\\>\"，其中 \"<\" 和 \">\" 是为了将作为前后缀的子词区分出来。而且，这里的子词 \"her\" 与整词 \"\\<her\\>\" 也可被区分。给定一个词 $w$， 通常我们的习惯于将字符长度在 3 到 6 之间的所有子词和特殊子词的并集 $\\mathcal{G}_w$ 取出。假设词典中任意子词 $g$ 的子词向量为 $\\boldsymbol{z}_g$ ，我们可以把使用负采样的 skip-gram 模型的损失函数\n",
    "$$\n",
    "-\\log P\\left(w_o \\mid w_c\\right)=-\\log \\frac{1}{1+\\exp \\left(-\\boldsymbol{u}_o^T \\boldsymbol{v}_c\\right)}-\\sum_{k=1, w_k \\sim P(w)}^K \\log \\frac{1}{1+\\exp \\left(\\boldsymbol{u}_{i_k}^T \\boldsymbol{v}_c\\right)}\n",
    "$$\n",
    "直接替换成\n",
    "$$\n",
    "\\begin{aligned}\n",
    "& -\\log P\\left(w_o \\mid w_c\\right)= \\\\\n",
    "& -\\log \\frac{1}{1+\\exp \\left(-\\boldsymbol{u}_o^T \\sum_{g \\in \\mathcal{G}_{w_c}} \\boldsymbol{z}_g\\right)}-\\sum_{k=1, w_k \\sim P(w)}^K \\log \\frac{1}{1+\\exp \\left(\\boldsymbol{u}_{i_k}^T \\sum_{g \\in \\mathcal{G}_{w_c}} \\boldsymbol{z}_g\\right)}\n",
    "\\end{aligned}\n",
    "$$\n",
    "不难看出，原中心词向量被替换成了中心词的子词向量之和，这与整词学习(word2vec, Glove)不同，并且词典以外的新词也可以使用fastText中相应的子词向量之和来表达。\n",
    "\n",
    "## 二、提出动机\n",
    "\n",
    "> Word2Vec的局限\n",
    "\n",
    "**Word2Vec不能处理训练中未出现的词(Out of Vocabulary, OOV)**\n",
    "\n",
    "例如：tensor, flow已经在Word2Vec词典出现，但tensorflow未出现过 --> OOV Error\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/5bd40eaba6ac491f9d655112ec5d09dae93bc22c0af94de78a4db6b45b950e59)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**无法处理形态相同的词（morphology)，即词根相同的词**\n",
    "\n",
    "对于具有相同词根(eat)的词， eaten，eating，eats，可能难以同时出现，实际训练就容易将其当成独一无二的词，也就是语义类似的词无法做到参数共享。\n",
    "$$\n",
    "Shared \\;\\;radical\\\\\n",
    "eat \\;\\; eat\\textcolor{red}s \\;\\; eat\\textcolor{red}{en} \\;\\; eat\\textcolor{red}{er} \\;\\; eat\\textcolor{red}{ing}\n",
    "$$\n",
    "\n",
    "> 为了解决以上问题，fastText提出\n",
    "\n",
    "## sub-word的产生\n",
    "\n",
    "**Step 1**：给定一个词，并且在词首尾添加上<>表示开始与结束\n",
    "\n",
    "\n",
    "$$\n",
    "eating\\;\\; →  \\;\\; \\textcolor{red}<eating\\textcolor{red}>\n",
    "$$\n",
    "\n",
    "\n",
    "**Step 2**：设定n-gram的滑动窗口大小 n=3（可以设置为其他值），对词进行滑动\n",
    "$$\n",
    "<eati\\textcolor{red}{ng>}\n",
    "$$\n",
    "**Step 3**：当n=3, 4, 5, 6时所得到的n-grams列表\n",
    "$$\n",
    "\\begin{aligned}\n",
    "&\\begin{array}{lll}\n",
    "Word & Length(n) & Character n-grams\\\\\n",
    "\\text { eating } & 3 & \\text { <ea, east, ati, tin, ing, ng> } \\\\\n",
    "\\hline \\text { eating } & 4 & \\text { <eat, eati, atin, ting, ing> } \\\\\n",
    "\\hline \\text { eating } & 5 & \\text { <eati, eatin, ating, ting> } \\\\\n",
    "\\hline \\text { eating } & 6 & \\text { <eatin, eating, ating> }\n",
    "\\end{array}\n",
    "\\end{aligned}\n",
    "$$\n",
    "**问题:这样会存在大量的唯一的n-grams**\n",
    "\n",
    "### Hash\n",
    "\n",
    "由于 n-gram 的量远比 word 大的多，完全存下所有的 n-gram 不太现实。fastText 采用的是 Hash 桶的方式，把所有的 n-gram 映射到 buckets 个桶中，而映射到相同桶的 n-gram 共享同一个 embedding vector，如下图所示\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/2b94d162901d4fc2ad809c534baef9a06b8c6bb4bfe54869bf8a81ec2d30d46f)\n",
    "\n",
    "图中 Win 代表整个 Embedding 矩阵，其中前 V 行是 word Embedding，后 Buckets 行是 n-gram Embedding，每个 n-gram 通过 hash 函数之后映射到 0~Bucket-1 位置，得到对应的 embedding 向量。用哈希的方式既能保证查找时 O (1) 的效率，又可能把内存消耗控制在 O (buckets * dim) 范围内。不过这种方法潜在的问题是存在哈希冲突，不同的 n-gram 可能会共享同一个 embedding。如果桶大小取的足够大，这种影响会很小\n",
    "\n",
    "> 具体实现过程为[负采样的Skip-Gram模型](https://aistudio.baidu.com/aistudio/projectdetail/5406915?contributionType=1&sUid=2554132&shared=1&ts=1678689283700)，唯一的区别就是对中心词的子词化的处理\n",
    "\n",
    "## 四、优劣势分析\n",
    "\n",
    "fastText 对于形态丰富的语言较重要，例如阿拉伯语、德语和俄语。例如，德语中有很多复合词，例如乒乓球（英文 table tennis）在德语中叫 \"Tischtennis\"。fastText 可以通过子词表达两个词的相关性，例如 \"Tischtennis\" 和 \"Tennis\"，特别是当训练预料规模较小的时候，提升尤为出色。\n",
    "$$\n",
    "\\begin{array}{|l|l|l|l|}\n",
    "\\hline & \\text { word2vec-skipgram } & \\text { word2vec-cbow } & \\text { fasttext } \\\\\n",
    "\\hline \\text { Czech } & 52.8 & 55.0 & \\mathbf{7 7 . 8} \\\\\n",
    "\\hline \\text { German } & 44.5 & 45.0 & \\mathbf{5 6 . 4} \\\\\n",
    "\\hline \\text { English } & 70.1 & 69.9 & \\mathbf{7 4 . 9} \\\\\n",
    "\\hline \\text { Italian } & 51.5 & 51.8 & \\mathbf{6 2 . 7} \\\\\n",
    "\\hline\n",
    "\\end{array}\n",
    "$$\n",
    "但与Word2Vec相比，FastText降低了**语义类比**的任务性能，随着语料库规模的增加，两者的差异越来越小。\n",
    "$$\n",
    "\\begin{array}{|c|c|c|c|}\n",
    "\\hline & \\text { word2vec-skipgram } & \\text { word2vec-cbow } & \\text { fasttext } \\\\\n",
    "\\hline \\text { Czech } & 25.7 & 27.6 & 27.5 \\\\\n",
    "\\hline \\text { German } & 66.5 & 66.8 & 62.3 \\\\\n",
    "\\hline \\text { English } & 78.5 & 78.2 & 77.8 \\\\\n",
    "\\hline \\text { Italian } & 52.3 & 54.7 & 52.3 \\\\\n",
    "\\hline\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "- fastText比常规的skipgram慢1.5倍，因为增加了n-grams的开销\n",
    "- 在单词相似度任务中，带有n-grams的sub-word信息比CBOW和Skip-gram基线具有更好的性能。\n",
    "\n",
    "另外，如果遇到一个新词，对于 fastText 来说，它可以从训练集中找出这个新词的所有子词向量，然后做个求和，就能算出这个新词的词向量了\n",
    "\n",
    "\n",
    "\n",
    "## 五、实践\n",
    "\n",
    "1.使用官方CLI工具\n",
    "\n",
    "[Word representations · fastText](https://fasttext.cc/docs/en/unsupervised-tutorial.html)\n",
    "\n",
    "2.使用gensim的fasttext\n",
    "\n",
    "[FastText Model — gensim (radimrehurek.com)](https://radimrehurek.com/gensim/auto_examples/tutorials/run_fasttext.html)\n",
    "\n",
    "3.使用PaddlePaddle进行简单的尝试！\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f111000d-d212-4f3a-a42a-4cbf2f9a3baa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:20:20.604909Z",
     "iopub.status.busy": "2024-07-16T06:20:20.604721Z",
     "iopub.status.idle": "2024-07-16T06:20:25.178374Z",
     "shell.execute_reply": "2024-07-16T06:20:25.178074Z",
     "shell.execute_reply.started": "2024-07-16T06:20:20.604897Z"
    }
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from sklearn.model_selection import train_test_split\n",
    " \n",
    "import pandas as pd\n",
    "\n",
    "data=pd.read_csv(\"data/ag_news/train.csv\",header=None,encoding=\"utf-8\")\n",
    "train,val=train_test_split(data,test_size=0.2,shuffle=True)\n",
    "# 我们这里只取新闻描述，直接使用验证集充当测试集，验证集不参与训练\n",
    "train_x=list(train.iloc[:,2])\n",
    "train_y=list(train.iloc[:,0])\n",
    "val_x=list(val.iloc[:,2])\n",
    "val_y=list(val.iloc[:,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c3add69e-ed8b-415e-a904-d2edb3d0345e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:20:29.913772Z",
     "iopub.status.busy": "2024-07-16T06:20:29.912580Z",
     "iopub.status.idle": "2024-07-16T06:20:29.921530Z",
     "shell.execute_reply": "2024-07-16T06:20:29.920479Z",
     "shell.execute_reply.started": "2024-07-16T06:20:29.913674Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HOUSTON - Marine researchers are hoping to prevent tides of destructive algae from killing fish and fouling beaches through early detection of the microscopic organisms. \n",
      "4\n"
     ]
    }
   ],
   "source": [
    "# 新闻描述\n",
    "print(train_x[1])\n",
    "# 类别\n",
    "print(train_y[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "270adae3-01d5-4d63-a76c-5aaaf9348b73",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:21:54.273966Z",
     "iopub.status.busy": "2024-07-16T06:21:54.273006Z",
     "iopub.status.idle": "2024-07-16T06:21:55.334216Z",
     "shell.execute_reply": "2024-07-16T06:21:55.333842Z",
     "shell.execute_reply.started": "2024-07-16T06:21:54.273908Z"
    }
   },
   "outputs": [],
   "source": [
    "# 数据清洗\n",
    "# 目的是保留每个句子中的所有单词与空格\n",
    "def clear(data):\n",
    "    for i, s in enumerate(data):\n",
    "        clears = \"\"\n",
    "        # 遍历当前句子每一个字符\n",
    "        for char in s:\n",
    "            # 若字符是字母或空格则保留\n",
    "            if(char.isalpha() or char == \" \"):\n",
    "                clears += char\n",
    "        data[i] = clears\n",
    "clear(train_x)\n",
    "clear(val_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1bebff28-d8de-4a06-ac3f-9b5872d1535d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:22:02.024684Z",
     "iopub.status.busy": "2024-07-16T06:22:02.023772Z",
     "iopub.status.idle": "2024-07-16T06:22:02.595289Z",
     "shell.execute_reply": "2024-07-16T06:22:02.594923Z",
     "shell.execute_reply.started": "2024-07-16T06:22:02.024625Z"
    }
   },
   "outputs": [],
   "source": [
    "#拼接所有句子，制作词典\n",
    "\n",
    "document = \"\"\n",
    "for s in train_x:\n",
    "    document += \" \" + str(s).lower()\n",
    "\n",
    "# 按照空格拆分,并进行简单的数据清洗\n",
    "clear_d = []\n",
    "for word in document.split(\" \"):\n",
    "    if(word.isalpha()):\n",
    "        clear_d.append(word)\n",
    "\n",
    "# Counter函数统计词频\n",
    "freq = dict(Counter(clear_d))\n",
    "idx2word = [\"<pad>\"] + list(freq.keys()) + [\"<unk>\"]\n",
    "word2idx = { w:idx for idx, w in enumerate(idx2word)}\n",
    "vocab_size = len(idx2word)\n",
    "\n",
    "del freq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1056d359-e2f8-4fea-8c85-742c5c333e27",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:22:09.637274Z",
     "iopub.status.busy": "2024-07-16T06:22:09.636256Z",
     "iopub.status.idle": "2024-07-16T06:22:10.447521Z",
     "shell.execute_reply": "2024-07-16T06:22:10.447291Z",
     "shell.execute_reply.started": "2024-07-16T06:22:09.637211Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "原来的句子\n",
      "The US military in Afghanistan dismissed a new videotape featuring Osama bin Laden as  quotpropaganda quot and insisted that bin Laden would be caught but it acknowledged having no fix on the AlQaeda leader s whereabouts\n",
      "They clean hotel rooms tend bars cook meals and wait tables at many of Bostons prestigious hotels and universities an army of mostly immigrant and lowwage workers that labors behind the scenes\n",
      "编码代替后的句子\n",
      "[28, 395, 1334, 64, 2855, 3583, 72, 73, 12201, 15765, 13599, 9099, 9100, 78, 38672, 769, 9, 9327, 25, 9099, 9100, 751, 764, 634, 230, 103, 6511, 3684, 604, 11941, 143, 28, 7491, 2875, 66, 13598]\n",
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 9, 18, 19, 20, 14, 21, 22, 9, 23, 24, 25, 26, 27, 28, 29]\n"
     ]
    }
   ],
   "source": [
    "def corpus_encode(corpus):\n",
    "    # 分词操作\n",
    "    for i in range(len(corpus)):\n",
    "        # 每个句子直接分词\n",
    "        s1 = corpus[i]\n",
    "        sentences = corpus[i].lower().split(\" \")\n",
    "\n",
    "        # 存储当前句子的编码信息\n",
    "        c = []\n",
    "        for word in sentences:\n",
    "            # 是单词才编码\n",
    "            if(word.isalpha()):\n",
    "                # 用词典中的索引代表该单词，若不在词典则用UNK的索引代表该单词\n",
    "                c.append(word2idx.get(word, word2idx[\"<unk>\"]))\n",
    "            # if(len(c)==0):\n",
    "                # print(\"原来的句子\", s1)\n",
    "                # print(\"分解以后的句子\", sentences)\n",
    "\n",
    "            # 编码代替原先的句子\n",
    "            corpus[i] = c\n",
    "            \n",
    "print(\"原来的句子\")\n",
    "print(val_x[0])\n",
    "print(train_x[0])\n",
    "corpus_encode(val_x)\n",
    "corpus_encode(train_x)\n",
    "print(\"编码代替后的句子\")\n",
    "print(val_x[0])\n",
    "print(train_x[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1ebd5e2f-1e82-43a2-a2a6-d33fe1376077",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:22:29.242576Z",
     "iopub.status.busy": "2024-07-16T06:22:29.241649Z",
     "iopub.status.idle": "2024-07-16T06:22:30.897141Z",
     "shell.execute_reply": "2024-07-16T06:22:30.896776Z",
     "shell.execute_reply.started": "2024-07-16T06:22:29.242524Z"
    }
   },
   "outputs": [],
   "source": [
    "# set集合，大括号中每组词向量为二维list\n",
    "Features = set()\n",
    "\n",
    "def word_N_gram(s, N = 2, train = False):\n",
    "    features = []\n",
    "    for i in range(len(s) - N + 1):\n",
    "        f = str(s[i : i+N])\n",
    "        if train:\n",
    "            Features.add(f) # 训练阶段应该统计有哪些特征\n",
    "        # 将一句话的每个词进一步拆分成多个小的词向量，例如like  --> li ik ke\n",
    "        features.append(f)\n",
    "    return features\n",
    "\n",
    "# 生成每个句子的n-gram特征\n",
    "train_n_gram = []\n",
    "for s in train_x:\n",
    "    # 已有的词向量与又拆分的二元词向量累加作为训练的词向量\n",
    "    train_n_gram.append(word_N_gram(s, train=True))\n",
    "val_n_gram = []\n",
    "for s in val_x:\n",
    "    val_n_gram.append(word_N_gram(s))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c0af7ce5-a734-43bd-bcf6-526b29bc25ad",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:22:34.587813Z",
     "iopub.status.busy": "2024-07-16T06:22:34.586969Z",
     "iopub.status.idle": "2024-07-16T06:22:34.600904Z",
     "shell.execute_reply": "2024-07-16T06:22:34.600036Z",
     "shell.execute_reply.started": "2024-07-16T06:22:34.587771Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_x) == len(train_n_gram)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "90ed8d1d-fbd3-4e8d-b71b-66a598c0c41d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:22:39.431909Z",
     "iopub.status.busy": "2024-07-16T06:22:39.431041Z",
     "iopub.status.idle": "2024-07-16T06:22:39.619025Z",
     "shell.execute_reply": "2024-07-16T06:22:39.618667Z",
     "shell.execute_reply.started": "2024-07-16T06:22:39.431854Z"
    }
   },
   "outputs": [],
   "source": [
    "# 给n-gram特征建立索引表\n",
    "idx2ngram = [\"[<pad>]\"] + list(Features) + [\"[<unk>]\"]\n",
    "ngram2idx = {w:c for c, w in enumerate(idx2ngram)}\n",
    "ngram_size = len(idx2ngram)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b4bb06c4-cd84-4035-a1b2-eb92584ae5a3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:22:43.761932Z",
     "iopub.status.busy": "2024-07-16T06:22:43.761163Z",
     "iopub.status.idle": "2024-07-16T06:22:44.801093Z",
     "shell.execute_reply": "2024-07-16T06:22:44.800761Z",
     "shell.execute_reply.started": "2024-07-16T06:22:43.761891Z"
    }
   },
   "outputs": [],
   "source": [
    "# 给特征进行编码\n",
    "def encode_gram(n_gram):\n",
    "    for i, s in enumerate(n_gram):\n",
    "        feature = []\n",
    "        for word in s:\n",
    "            feature.append(ngram2idx.get(word, ngram2idx[\"[<unk>]\"]))\n",
    "        n_gram[i] = feature\n",
    "encode_gram(train_n_gram)\n",
    "encode_gram(val_n_gram)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "87e76ca1-278a-45ff-bb3f-e8c5e18634ea",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:22:51.140271Z",
     "iopub.status.busy": "2024-07-16T06:22:51.139811Z",
     "iopub.status.idle": "2024-07-16T06:22:53.142361Z",
     "shell.execute_reply": "2024-07-16T06:22:53.141984Z",
     "shell.execute_reply.started": "2024-07-16T06:22:51.140245Z"
    }
   },
   "outputs": [],
   "source": [
    "import paddle\n",
    " \n",
    "# from torch.nn.utils.rnn import pad_sequence\n",
    "from paddle.io import DataLoader \n",
    "from paddle.io import Dataset\n",
    "# import paddle.static.nn as F\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e820559a-68b4-4602-8275-cda52c9d88c7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:22:57.947227Z",
     "iopub.status.busy": "2024-07-16T06:22:57.945193Z",
     "iopub.status.idle": "2024-07-16T06:22:58.128086Z",
     "shell.execute_reply": "2024-07-16T06:22:58.127754Z",
     "shell.execute_reply.started": "2024-07-16T06:22:57.947144Z"
    }
   },
   "outputs": [],
   "source": [
    "# 统计每个句子的长度，为了后面取均值\n",
    "\n",
    "# map(function, iterable, ...)\n",
    "lensTrianWord = list(map(len, train_x))\n",
    "lensVarWord = paddle.to_tensor(list(map(len, val_x)))\n",
    "\n",
    "lensTrianNGram = list(map(len, train_n_gram))\n",
    "lenValNGram = paddle.to_tensor(list(map(len, val_n_gram)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9291fb07-4339-4a9c-81db-c9437110fd75",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:05.185478Z",
     "iopub.status.busy": "2024-07-16T06:23:05.184900Z",
     "iopub.status.idle": "2024-07-16T06:23:10.595041Z",
     "shell.execute_reply": "2024-07-16T06:23:10.594212Z",
     "shell.execute_reply.started": "2024-07-16T06:23:05.185444Z"
    }
   },
   "outputs": [],
   "source": [
    "# 句子进行pad,这种方式即全以最长的句子为规模进行组装\n",
    "def padding(train_x, max_len):\n",
    "    for i in range(0, len(train_x)):\n",
    "        zero_list = [0 for i in range(0, max_len - len(train_x[i]))]\n",
    "        train_x[i].extend(zero_list)\n",
    "\n",
    "\n",
    "# padding(train_x, 168)\n",
    "# padding(train_n_gram, 167)\n",
    "padding(train_x, 173)\n",
    "padding(train_n_gram, 172)\n",
    "padding(val_x, 173)\n",
    "padding(val_n_gram, 172)\n",
    "\n",
    "\n",
    "train_x = list(map(lambda x : paddle.to_tensor(x), train_x))\n",
    "val_x = list(map(lambda x : paddle.to_tensor(x), val_x))\n",
    "\n",
    "train_n_gram = list(map(lambda x : paddle.to_tensor(x), train_n_gram))\n",
    "val_n_gram = list(map(lambda x : paddle.to_tensor(x), val_n_gram))\n",
    "\n",
    "# 原本类别是1 2 3 4\n",
    "# CrossEntropyLoss()的类别要求0 1 2 3 这种格式，因此-1\n",
    "# train_y最终会放入数据生成器中，因此写列表元素为tensor形式\n",
    "train_y = list(map(lambda x : paddle.to_tensor(x) - 1, train_y))\n",
    "val_y = list(map(lambda x : x - 1, val_y))\n",
    "val_y = paddle.to_tensor(val_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "35f7b334-a16c-42ef-9d84-0ab01f848442",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:17.986269Z",
     "iopub.status.busy": "2024-07-16T06:23:17.985091Z",
     "iopub.status.idle": "2024-07-16T06:23:18.007600Z",
     "shell.execute_reply": "2024-07-16T06:23:18.006702Z",
     "shell.execute_reply.started": "2024-07-16T06:23:17.986197Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Tensor(shape=[1], dtype=int64, place=Place(cpu), stop_gradient=True,\n",
       "       [36])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lensVarWord[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c68d0c32-91a7-40df-af07-3c0f1b0e84b7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:23.999597Z",
     "iopub.status.busy": "2024-07-16T06:23:23.998717Z",
     "iopub.status.idle": "2024-07-16T06:23:24.011178Z",
     "shell.execute_reply": "2024-07-16T06:23:24.010215Z",
     "shell.execute_reply.started": "2024-07-16T06:23:23.999534Z"
    }
   },
   "outputs": [],
   "source": [
    "class mydata(Dataset):\n",
    "    def __init__(self, train_x, train_Ngram, train_y, lensX, lensNGram):\n",
    "        super(mydata, self).__init__()\n",
    "        self.train_x = train_x\n",
    "        self.train_Ngram = train_Ngram\n",
    "        self.train_y = train_y\n",
    "        self.lensX = lensX\n",
    "        self.lensNGram = lensNGram\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.train_x)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.train_x[idx], self.train_Ngram[idx], self.lensX[idx], self.lensNGram[idx], self.train_y[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "77d86d24-f1a0-47a8-890d-04f20c2a1bd7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:29.602680Z",
     "iopub.status.busy": "2024-07-16T06:23:29.601794Z",
     "iopub.status.idle": "2024-07-16T06:23:29.637886Z",
     "shell.execute_reply": "2024-07-16T06:23:29.637219Z",
     "shell.execute_reply.started": "2024-07-16T06:23:29.602633Z"
    }
   },
   "outputs": [],
   "source": [
    "batch_size = 20\n",
    "data = mydata(train_x, train_n_gram, train_y, lensTrianWord, lensTrianNGram)\n",
    "dataloader = DataLoader(data, batch_size=batch_size, shuffle=True, num_workers=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9f9fc4d3-06fa-40d6-947a-c1620cddee77",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:36.067133Z",
     "iopub.status.busy": "2024-07-16T06:23:36.065340Z",
     "iopub.status.idle": "2024-07-16T06:23:36.117282Z",
     "shell.execute_reply": "2024-07-16T06:23:36.116497Z",
     "shell.execute_reply.started": "2024-07-16T06:23:36.067081Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20, 173] [20, 172] [20] [20] [20, 1]\n"
     ]
    }
   ],
   "source": [
    "for x,ngram,lenx,lenNgram,y in dataloader:\n",
    "    print(x.shape,ngram.shape,lenx.shape,lenNgram.shape,y.shape)\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "160023da-7386-43f5-b5a5-37527e64f0c7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:41.268242Z",
     "iopub.status.busy": "2024-07-16T06:23:41.267985Z",
     "iopub.status.idle": "2024-07-16T06:23:41.270913Z",
     "shell.execute_reply": "2024-07-16T06:23:41.270546Z",
     "shell.execute_reply.started": "2024-07-16T06:23:41.268218Z"
    }
   },
   "outputs": [],
   "source": [
    "import paddle.nn as nn\n",
    "\n",
    "from paddle.nn import Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "601a3f5f-24ed-41c7-8919-79ca1f729e50",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:45.949057Z",
     "iopub.status.busy": "2024-07-16T06:23:45.948154Z",
     "iopub.status.idle": "2024-07-16T06:23:45.955943Z",
     "shell.execute_reply": "2024-07-16T06:23:45.954474Z",
     "shell.execute_reply.started": "2024-07-16T06:23:45.949005Z"
    }
   },
   "outputs": [],
   "source": [
    "# 模型深度\n",
    "d_model = 50\n",
    "# 类别数量\n",
    "class_num = 4\n",
    "ngram_size = len(ngram2idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1d273396-60d3-438e-b071-a91fa17d820d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:50.745144Z",
     "iopub.status.busy": "2024-07-16T06:23:50.744126Z",
     "iopub.status.idle": "2024-07-16T06:23:50.759640Z",
     "shell.execute_reply": "2024-07-16T06:23:50.758817Z",
     "shell.execute_reply.started": "2024-07-16T06:23:50.745088Z"
    }
   },
   "outputs": [],
   "source": [
    "class cbow(Layer):\n",
    "    def __init__(self, vocab_size, ngram_size, d_model, class_num) -> None:\n",
    "        super(cbow, self).__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.d_model = d_model\n",
    "        self.class_num = class_num\n",
    "        # torch.nn.Embedding(num_embeddings, embedding_dim, padding_idx=None)\n",
    "        # num_embeddings (int) – size of the dictionary of embeddings\n",
    "        # padding_idx 指定pad的索引，指定之后该索引不会随着训练的更新而更新\n",
    "        self.embed1 = nn.Embedding(vocab_size, d_model, padding_idx=0)\n",
    "        self.embed2 = nn.Embedding(ngram_size, d_model, padding_idx=0)\n",
    "        self.linear = nn.Linear(d_model, class_num)\n",
    "\n",
    "\n",
    "    def forward(self, x, ngram, lensX, lenNgram):\n",
    "        # x[batch, maxlen1], ngram[batch, maxlen2], lensX[batch], lenNgram[batch]\n",
    "        # Embedding的参数维度通常是放在bts之后，与Embedding扩展维度之前\n",
    "        x = self.embed1(x) # x[batch, maxlen1, d_model]\n",
    "        ngram = self.embed2(ngram) # ngram[batch, maxlen1, d_model]\n",
    "        # 词向量求和\n",
    "        x = paddle.sum(x, axis=1) # x[batch, d_model]\n",
    "        ngram = paddle.sum(ngram, axis=1) # ngram[batch, d_model]\n",
    "        x = x + ngram\n",
    "        lens = lensX + lenNgram\n",
    "        lens = lens.unsqueeze(1) # lens[batch, 1]\n",
    "        # 取均值\n",
    "        x /= lens\n",
    "        output = self.linear(x)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7e49d9ec-edc9-4e19-95ff-3ce98403af92",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:23:56.412820Z",
     "iopub.status.busy": "2024-07-16T06:23:56.412654Z",
     "iopub.status.idle": "2024-07-16T06:23:56.690625Z",
     "shell.execute_reply": "2024-07-16T06:23:56.690344Z",
     "shell.execute_reply.started": "2024-07-16T06:23:56.412807Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/arcment/miniconda3/envs/pnlp/lib/python3.8/site-packages/paddle/fluid/dygraph/math_op_patch.py:275: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.int64, the right dtype will convert to paddle.float32\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[20, 4]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model=cbow(vocab_size,ngram_size,d_model,class_num)\n",
    "model(x,ngram,lenx,lenNgram).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "baceebba-5a68-498b-8dbb-89b37cbe620c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:24:02.028390Z",
     "iopub.status.busy": "2024-07-16T06:24:02.026410Z",
     "iopub.status.idle": "2024-07-16T06:24:02.040717Z",
     "shell.execute_reply": "2024-07-16T06:24:02.039551Z",
     "shell.execute_reply.started": "2024-07-16T06:24:02.028323Z"
    }
   },
   "outputs": [],
   "source": [
    "from paddle.optimizer import Adagrad\n",
    "\n",
    "epochs = 5\n",
    "lr = 0.01\n",
    "\n",
    "optimize = Adagrad(parameters=model.parameters(), learning_rate=lr)\n",
    "lossCul = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c2e72727-d050-45db-9c98-132d539eee54",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-07-16T06:26:30.197792Z",
     "iopub.status.busy": "2024-07-16T06:26:30.197151Z",
     "iopub.status.idle": "2024-07-16T06:31:43.224155Z",
     "shell.execute_reply": "2024-07-16T06:31:43.223152Z",
     "shell.execute_reply.started": "2024-07-16T06:26:30.197757Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/arcment/miniconda3/envs/pnlp/lib/python3.8/site-packages/paddle/fluid/dygraph/math_op_patch.py:275: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.int64, the right dtype will convert to paddle.float32\n",
      "  warnings.warn(\n",
      "/Users/arcment/miniconda3/envs/pnlp/lib/python3.8/site-packages/paddle/fluid/framework.py:3979: DeprecationWarning: Op `adagrad` is executed through `append_op` under the dynamic mode, the corresponding API implementation needs to be upgraded to using `_C_ops` method.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epochs: 1  iter: 1000  loss: 0.4571259021759033\n",
      "epochs: 1  iter: 2000  loss: 0.43689122796058655\n",
      "epochs: 1  iter: 3000  loss: 0.41904374957084656\n",
      "epochs: 1  iter: 4000  loss: 0.40534794330596924\n",
      "epochs: 2  iter: 1000  loss: 0.29361870884895325\n",
      "epochs: 2  iter: 2000  loss: 0.28906679153442383\n",
      "epochs: 2  iter: 3000  loss: 0.28381845355033875\n",
      "epochs: 2  iter: 4000  loss: 0.2811451852321625\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(epochs):\n",
    "    #总损失\n",
    "    allloss=0\n",
    "    for step,(x,ngram,lenx,lenNgram,y) in enumerate(dataloader):\n",
    "        output=model(x,ngram,lenx,lenNgram)\n",
    "        #计算极大似然函数损失\n",
    "        loss=lossCul(output,y)\n",
    "        optimize.clear_grad()\n",
    "        loss.backward()\n",
    "        optimize.step()\n",
    "        \n",
    "        allloss+=loss\n",
    "        if (step+1) % 1000==0:\n",
    "            print(\"epochs:\", epoch+1, \" iter:\", step+1, \" loss:\", (allloss / (step+1)).item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ffaa577-fbe1-4941-ab64-b65f3e3749de",
   "metadata": {},
   "source": [
    "# 五、总结\n",
    "\n",
    "## Fasttext模型优缺点\n",
    "\n",
    "**优点：**\n",
    "1. 速度非常快，并且效果还可以。\n",
    "2. 有开源实现，可以快速上手使用。\n",
    "\n",
    "**缺点：**\n",
    "1. 模型结构简单，所以目前来说，不是最优的模型。\n",
    "2. 因为使用词袋思想，所以语义信息获取有限。\n",
    "\n",
    "## 论文总结\n",
    "\n",
    "**关键点：**\n",
    "1. 基于深度学习的文本分类方法效果好，但是速度比较慢\n",
    "2. 基于线性分类器的机器学习方法效果还行，速度也比较快，但是需要做烦琐的特征工程\n",
    "3. Fasttext模型的提出\n",
    "\n",
    "**创新点：**\n",
    "1. 提出了一种新的文本分类模型—Fasttext模型\n",
    "2. 利用了一些加快文本分类和使得文本分类效果更好的技巧——层次softmax和n-gram特征。\n",
    "3. 在文本分类和tag预测两个任务上得到了又快又好的结果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a01d427-8684-4a0e-b64f-55f5586484ba",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pnlp",
   "language": "python",
   "name": "pnlp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
